Import libraries¶
In [1]:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression,Lasso,Ridge,LassoCV
Load and Review data¶
In [2]:
House_data = pd.read_csv("../data/kc_house_data.csv",engine='python')
In [3]:
House_data.shape
Out[3]:
In [4]:
House_data.head(5)
Out[4]:
In [5]:
House_data.info()
In [6]:
House_data.describe()
Out[6]:
In [7]:
House_data.bedrooms.value_counts()
Out[7]:
In [8]:
House_data.floors.value_counts()
Out[8]:
In [9]:
House_data.view.value_counts()
Out[9]:
In [10]:
House_data.waterfront.value_counts()
Out[10]:
In [11]:
House_data.condition.value_counts()
Out[11]:
In [12]:
House_data.grade.value_counts()
Out[12]:
Observations¶
- The maximum number of bedrooms in a house are 33. So might wanted to look at that record and check if it is an outlier.
- one floor houses are the most common type of houses sold
- Very few houses have view to a waterfront and these houses might be costly
- House condition is rated from 1 to 5 and the most common rating is 3
- House grade is rated from 1 to 13 and the most common rating is 7
Converting day hours to date time object¶
In [13]:
House_data['date'] = pd.to_datetime(House_data['date']).dt.to_period('m')
House_data.head()
Out[13]:
To checlk if there are any null values or missing data¶
In [14]:
House_data.isnull().any()
Out[14]:
There are no null values or missing values in the data. No data cleansing is required.
To check if there are duplicate entries¶
In [15]:
House_data.duplicated().sum()
Out[15]:
Univariate analysis¶
Plotted the box plot of different columns to check if there are any outliers¶
In [16]:
for i in House_data.iloc[:, 2:].columns:
House_data.iloc[:, 1:].boxplot(column=i)
plt.show()
To see the # of houses sold based on number of bedrooms¶
In [17]:
House_data['bedrooms'].value_counts().plot(kind='bar')
plt.title('number of Bedrooms')
plt.xlabel('Bedrooms')
plt.ylabel('Count')
sns.despine
Out[17]:
The most common type of house sold are the ones with three bedrooms. So this is helpful as we can understand 3 bedroom houses have more demand followed by 4 bedroom. So # of bedrooms might be an important factor while we fix the price.
To check if location (lattitude and longitude have an impact on # of houses sold¶
In [18]:
plt.figure(figsize=(10,10))
sns.jointplot(x=House_data.lat.values, y=House_data.long.values, height=10)
plt.ylabel('Longitude', fontsize=12)
plt.xlabel('Latitude', fontsize=12)
plt.show()
sns.despine
Out[18]:
Observation¶
we see large number of houses between latitudes 47.5 and 47.8 and in terms of longitude there are large number of houses between -122.2 and -122.4. This location might be the ideal location for people to live and house prices might vary based up on this lattitude and longitude
Bivariate analysis¶
In [19]:
HData_attr = House_data.iloc[:, 0:21]
sns.pairplot(HData_attr, diag_kind='kde')
Out[19]:
In [20]:
house_corr = House_data.corr(method ='pearson')
house_corr
Out[20]:
Observations from bivariate analysis¶
- sqft_living, bathrooms, grade, sqft_above, sqft_living15 are the metrics which have high correlation to the target variable price. So these metrics might pop up as important metrics from the regression execrise
- Also few of the above highlighgted variables are also correlated with each other because of which only few of them might have significant impact on price because of multi-collinearity
In [21]:
plt.figure(figsize=(15,15))
corr = House_data.apply(lambda x: pd.factorize(x)[0]).corr()
ax = sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns,
linewidths=.2, cmap="YlGnBu")
Data pre processing¶
In [22]:
House_data.columns
Out[22]:
Defining the independent and dependent variables¶
In [23]:
X = House_data.drop(['price','id','date', 'zipcode', 'yr_renovated'],axis=1)
y = House_data[['price']]
splitting the data into train and test¶
In [24]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=110)
Linear regression¶
In [25]:
regression_model = LinearRegression()
regression_model.fit(X_train, y_train)
Out[25]:
In [26]:
for idx, col_name in enumerate(X_train.columns):
print("The coefficient for {} is {}".format(col_name, regression_model.coef_[0][idx]))
In [27]:
intercept = regression_model.intercept_[0]
print("The intercept for our model is {}".format(intercept))
Ridge regression¶
In [52]:
ridge = Ridge(alpha=.3, normalize=True)
ridge.fit(X_train,y_train)
print ("Ridge model:", (ridge.coef_))
Lasso regression¶
In [57]:
from sklearn.metrics import r2_score, get_scorer
from sklearn.linear_model import Lasso, Ridge, LassoCV,LinearRegression
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.model_selection import KFold, RepeatedKFold, GridSearchCV, cross_validate, train_test_split
In [58]:
cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)
lasso_alphas = np.linspace(0, 0.2, 21)
lasso = Lasso()
grid = dict()
grid['alpha'] = lasso_alphas
gscv = GridSearchCV(lasso,
grid,
scoring='neg_mean_absolute_error',
cv=cv,
n_jobs=-1)
results = gscv.fit(X_train, y_train)
print('MAE: %.5f' % results.best_score_)
print('Config: %s' % results.best_params_)
In [59]:
lasso = Lasso(alpha=0.2, normalize=True)
lasso.fit(X_train,y_train)
print ("Lasso model:", (lasso.coef_))
In [54]:
print(regression_model.score(X_train, y_train))
print(regression_model.score(X_test, y_test))
In [55]:
print(ridge.score(X_train, y_train))
print(ridge.score(X_test, y_test))
In [60]:
print(lasso.score(X_train, y_train))
print(lasso.score(X_test, y_test))
There is no considerable increase in the Rsquare value when Ridge or Lasso regression are used instead of linear regression
Decision Tree Regression¶
In [33]:
from sklearn.tree import DecisionTreeRegressor
from sklearn import tree
from sklearn.metrics import r2_score
from sklearn import metrics
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import GridSearchCV
In [34]:
dtree_up = DecisionTreeRegressor()
dtree_up.fit(X_train, y_train) # Fitting model with x_train and y_train
dtree_pred_up = dtree_up.predict(X_test) # Predicting the results
print('RMSE:', np.sqrt(mean_squared_error(y_test, dtree_pred_up, squared=False)))
print('r2 score: %.2f' % r2_score(y_test, dtree_pred_up))
print("Accuracy :",dtree_up.score(X_test, y_test))
Hyperparameter tuned Decision Tree Regression¶
In [35]:
dtree = DecisionTreeRegressor(random_state=5)
d=np.arange(1,21,1)
hyperParam = [{'max_depth':d}]
gsv = GridSearchCV(dtree,hyperParam,cv=5,verbose=1)
best_model = gsv.fit(X_train, y_train) # Fitting model with xtrain_scaler and y_train
dtree_pred_mms = best_model.best_estimator_.predict(X_test) # Predicting the results
print("Best HyperParameter: ",gsv.best_params_)
print('RMSE:', np.sqrt(mean_squared_error(y_test, dtree_pred_mms, squared=False)))
print('r2 score: %.2f' % r2_score(y_test, dtree_pred_mms))
print("Accuracy :",best_model.score(X_test, y_test))
Random forest Regressor¶
In [36]:
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import GridSearchCV
from sklearn import metrics
In [37]:
rf = RandomForestRegressor()
rf.fit(X_train, y_train) # Fitting model with x_train and y_train
rf_pred = rf.predict(X_test) # Predicting the results
print('RMSE:', np.sqrt(mean_squared_error(y_test, rf_pred, squared=False)))
print('r2 score: %.2f' % r2_score(y_test, rf_pred))
print("Accuracy :",rf.score(X_test, y_test))
Hyperparameter tuned Random forest Regressor¶
In [38]:
nEstimator = [140,160,180,200,220]
depth = [10,15,20,25,30]
RF = RandomForestRegressor()
hyperParam = [{'n_estimators':nEstimator,'max_depth': depth}]
gsv = GridSearchCV(RF,hyperParam,cv=5,verbose=1,scoring='r2',n_jobs=-1)
gsv.fit(X_train, y_train)
print("Best HyperParameter: ",gsv.best_params_)
scores = gsv.cv_results_['mean_test_score'].reshape(len(nEstimator),len(depth))
maxDepth=gsv.best_params_['max_depth']
nEstimators=gsv.best_params_['n_estimators']
model = RandomForestRegressor(n_estimators = nEstimators,max_depth=maxDepth)
model.fit(X_train, y_train) # Fitting model with x_train and y_train
# Predicting the results:
rf_pred_tune = model.predict(X_test)
print('RMSE:', np.sqrt(mean_squared_error(y_test, rf_pred_tune, squared=False)))
print('r2 score: %.2f' % r2_score(y_test, rf_pred_tune))
print("Accuracy :",model.score(X_test, y_test))
Gradient Boosting Regressor¶
In [39]:
from sklearn import ensemble
clf = ensemble.GradientBoostingRegressor(n_estimators = 400, max_depth = 5, min_samples_split = 2,
learning_rate = 0.1, loss = 'ls')
In [44]:
ensemble.GradientBoostingRegressor?
In [40]:
clf.fit(X_train, y_train)
Out[40]:
In [41]:
y_predict = clf.predict(X_test)
y_predict
Out[41]:
In [42]:
clf.score(X_test,y_test)
Out[42]:
In [46]:
clf.score?
In [45]:
gb_pred = clf.predict(X_test) # Predicting the results
print('RMSE:', np.sqrt(mean_squared_error(y_test, gb_pred, squared=False)))
print('r2 score: %.2f' % r2_score(y_test, gb_pred))
print("Accuracy :",clf.score(X_test, y_test))
Evaluation of different models (Comparison of different models and performance tuning)¶
- Linear regression, Lasso regression, Ridge regression, Decision tree regressor, Hyperparameter tuned decision tree regressor, Random forest regressor, Hyperparameter tuned random forest regressor and Gradient boosting regressor are the different models we tried for this problem.
- The Rsquare value that we achieved with linear regression is 70%
- To improve this we tried Lasso and Ridge regression, however in both these cases the Rsquare value obtained was 70%. No improvement in accuracy with respect to linear regression.
- In Lasso not many coefficients are going to zero implying that there is predictive power in many different variables and not concentrated in few variables
- Then we tried decision tree and we are able to improve the Rsquare value to 72%
- Then we tried decision tree with hyperparameter tuning by using Gridsearch and were able to improve the Rsquare value to 75%
- Then we tried random forest regressor and was able to increase the Rsquare to 87%
- Then we tried random forest with hyperparameter tuning by using Gridsearch and we found no significant improvement from 88%
- Finally we tried gradient boosting regressor and were able to achieve R square value of 90%
- Using grid search in our models we varied different hyperparameters and obtained the optimal hyperparameters.
Conclusion¶
- To predict the prices of houses we first analysed different independent variables by seeing the correlation of these variables with the target variable price
- We also looked at the multi-collinearity of this independent variables
- We then applied multi variate linear regression and achieved a test accuracy of 70%
- We then applied a large group of models and the best accuracy obtained from those models is from gradient boosting regressor which is 88%
- It is once again observed that ensemble techniques help us in achieving high accuracy either it is regression or classification as it involves a large group models and best of these models is taken